Systems and methods for capturing, displaying, and manipulating medical images and videos

ABSTRACT

A surgical image capture and display system includes a handheld image capture and pointing device and a display assembly. An image is captured by an image sensor of the handheld device and displayed on the display assembly. The image sensor detects light emitted by one or more beacons of the display assembly. The system determines, based on the light emitted by the one or more beacons, a position or orientation of the handheld device relative to the display assembly. The system updates display of a graphical user interface comprising the image on the display assembly in accordance with the determined position or orientation of the handheld device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.17/507,712, filed Oct. 21, 2021, which claims the benefit of U.S.Provisional Application No. 63/104,266, filed Oct. 22, 2020, the entirecontents of which are hereby incorporated by reference herein.

FIELD

The present disclosure relates generally to medical imaging systems, andmore specifically systems and methods for capturing, displaying, andmanipulating medical images and videos.

BACKGROUND

Capturing and displaying medical and surgical images is integral to manymedical and surgical procedures. According to known techniques, medicalimages and videos may be captured by image sensors of one or morecameras and then displayed on computer monitors or dedicated displayscreens. According to known techniques, medical images and videos may bemanipulated using a mouse, keyboard, or touch-screen controls.

SUMMARY

As explained above, according to known techniques, medical images andvideos are captured by image sensors of various cameras, displayed oncomputer monitors or dedicated display screens, and manipulated using amouse, keyboard, or touch-screen controls. However, known techniques forcapturing, displaying, and manipulating medical images and videos arecumbersome and inconvenient for surgeons and other medicalpractitioners. For example, a practitioner who wishes to capture one ormore images or video frames and then display and manipulate the imagemay be required to use a camera to capture the image/video, then putdown the camera and use another input device such as a mouse, keyboard,or touch-screen display to manipulate the captured image. Therequirement to use a different device to capture the image and tomanipulate the captured image introduces delay and the potential forcontamination between the various devices. Accordingly, there is a needfor improved systems and methods for capturing, displaying, andmanipulating medical images and video, particularly methods and systemsthat obviate the need for a practitioner to use multiple differentdevices to capture images/video and to manipulate the capturedimages/video.

Disclosed herein are systems, methods, and techniques for capturing,displaying, and manipulating medical images and videos. A handheld imagecapture and pointing device may include one or more image sensorsconfigured to capture both white light video/images and infraredvideo/images. (As used herein, “infrared” or “IR” may refer to infraredand/or near-infrared (“NIR”) light.) The handheld image capture andpointing device may be aimed at a region of tissue of a patient and theone or more image sensors may capture a series of video frames in whitelight and a series of video frames in infrared light. An image from oneof the captured video frames may be displayed on a display of a displayassembly, wherein the display assembly comprises one or more infraredbeacons positioned proximate to the display assembly and configured toemit infrared light. After capturing the image with the handheld device,and while the image is displayed on the display assembly, the operatormay aim the handheld device at the display assembly such that theinfrared beacons are within the field of vision of the handheld deviceand such that the infrared light emitted by the beacons is detected byone or more of the image sensors of the device. Based on the infraredlight detected by the device, a processor associated with the system maydetermine a position and/or orientation of the handheld device relativeto the display assembly, and the system may cause a graphical userinterface including the displayed image to be updated in accordance withthe determined position and/or orientation. For example, the graphicaluser interface may be updated by annotating the displayed image, zoomingthe displayed image or scaling the size of an annotation on the image,rotating the displayed image or an annotation on the image, or the like.Thus, the user of the handheld image capture and pointing device may beable to use a single device both to capture medical images and tomanipulate the images while they are displayed, obviating the need tofrequently switch between different devices during medical procedures.

Furthermore, known techniques for extracting information from medicalvideo streams require manual input and are imprecise and cumbersome. Forexample, an IR video stream may have little visible information when afluorescence agent is not present in the vasculature of the tissuedepicted, and it may therefore be difficult for a system or user toselect a region of a frame of the IR video stream on which to focusbefore the arrival of the fluorescence agent. Accordingly, there is aneed for improved systems for extracting information from medical videostreams, including for selecting regions of frames of medical videostreams on which image analysis and/or visualization should beperformed.

Disclosed herein are systems, methods, and techniques for usingframe-synchronized object tracking to identify regions of medical videostreams for analysis and/or visualization. A system for medical imagecapture, display, manipulation, annotation, and analysis may beconfigured such that two (or more) video streams depicting the sametarget tissue may be simultaneously captured, wherein a first one of thevideo streams captures the target tissue in a first spectrum (e.g.,white light) and a second one of the video streams captures the targettissue in a second spectrum (e.g., IR). An object such as a tissuefeature may be tracked in one video stream (e.g., a white light videostream) to determine a location of the other video stream (e.g., an IRvideo stream) at which image analysis should be performed or from whichdata should be extracted. In this manner, a system may be configured touse frame-synchronized video channels (e.g., video streams) to extractinformation out of a first channel of video (e.g., a white light videochannel) and apply that information to corresponding positions in thesecond channel of video (e.g., an IR video channel).

Furthermore, known techniques for annotating and tracking tissue samplescomprise multiple manual steps and are error prone. During a medicalprocedure, a surgeon or other medical practitioner may take tissuesamples from various parts of the anatomy. These tissue samples may thenbe sent to pathology for analysis. After the analysis reports areobtained, the surgeon then has to correlate multiple pathology resultsto the tissue samples that were taken. This process is done manually andis prone to errors. Accordingly, there is a need for improved systemsand methods for marking different regions of anatomy with labels duringthe procedure, saving those annotations as part of video, and trackingpathology results associated with those regions of anatomy.

Disclosed herein are systems, methods, and techniques for capturingmedical images and generating annotation data to be stored inassociation with a region of interest in the medical image and/or inassociation with a tissue sample extracted from the tissue at an areaindicated by the region of interest in the image. A system may capture amedical image (e.g., a frame from a medical video) using a handheldimage capture and pointing device, display said medical image on adisplay assembly, determine a location in said image displayed on saiddisplay assembly at which the handheld device is pointed, and generateand store annotation data associated with a region of interest definedin the image at the location at which the handheld device is pointed.

According to an aspect, a surgical image capture and display system isprovided, the surgical image capture and display system comprising: ahandheld image capture and pointing device, the handheld devicecomprising an image sensor configured to detect white light and infraredlight; a display assembly configured to display one or more imagescaptured by the handheld device; and one or more processors configuredto cause the system to: capture, by the image sensor of the handhelddevice, an image; display, by a display of the display assembly, agraphical user interface comprising the image; while the graphical userinterface is displayed by the display, detect, by the image sensor,light emitted by one or more beacons of the display assembly; determine,based on the light emitted by the one or more beacons of the displayassembly, a position or orientation of the handheld device relative tothe display assembly; and update display of the graphical user interfacecomprising the image in accordance with the position or orientation ofthe handheld device.

Optionally, determining the position or orientation comprisesdetermining a distance from the display assembly to the handheld device.

Optionally, determining the position or orientation comprisesdetermining an offset angle of the handheld device with respect to aplane intersecting the display assembly at a right angle along a linefrom the top center of the display assembly to the bottom center of thedisplay assembly.

Optionally, determining the position or orientation comprisesdetermining both the position and the orientation of the handhelddevice.

Optionally, determining the position or orientation comprisesdetermining a location on the display at which the handheld device isaimed.

Optionally, determining the orientation comprises determining arotational orientation of the handheld device with respect to a linealong which the handheld device is aimed.

Optionally, capturing the image is performed prior to detecting thelight emitted by the one or more beacons.

Optionally, the system is configured to operate in an image capture modein which the image is captured and stored by the system; and the systemis configured to operate in an image manipulation mode in which theimage is displayed on the display and the graphical user interfaceaccepts one or more inputs for manipulation of the image, wherein theone or more inputs are executed via the handheld device and based atleast in part on the determined position or orientation of the handhelddevice.

Optionally, updating the graphical user interface comprises displaying acursor on the graphical user interface at a location on the display atwhich the handheld device is determined to be aimed.

Optionally, updating the graphical user interface comprises generatingand displaying an annotation on the displayed image at a location on thedisplay at which the handheld device is determined to be aimed.

Optionally, updating the graphical user interface comprises rotating animage annotation in accordance with the orientation of the handhelddevice.

Optionally, updating the graphical user interface comprises modifying azoom level at which the image is displayed in accordance with theposition or orientation of the handheld device.

Optionally, updating the graphical user interface comprises modifying asize of an image annotation in accordance with the position ororientation of the handheld device.

Optionally, updating the graphical user interface comprises setting afont size for the graphical user interface in accordance with theposition or orientation of the handheld device.

Optionally, updating the graphical user interface comprises displaying anotification based on an offset angle determined in accordance with thedetermined position or orientation of the handheld device.

Optionally, the one or more processors are configured to cause thesystem to: in accordance with the determined position or orientation ofthe handheld device, identify a first region of interest of the image;and apply one or more of a tracking algorithm and an image analysisalgorithm to the first region of interest of the image.

Optionally, the image is captured by the image capture and pointingdevice in a first spectrum; and the one or more processors areconfigured to cause the system to: capture, by the image capture andpointing device, a second image in a second spectrum; and apply one ormore of the tracking algorithm and the image analysis algorithm to asecond region of interest corresponding to the first region of interest,wherein the second region of interest is in the second image.

Optionally, determining the position or orientation of the handhelddevice is further based on data received from a sensor onboard thehandheld device, wherein the sensor comprises one or more of anaccelerometer, a gyroscope, and a compass.

Optionally, determining the position or orientation of the handhelddevice is further based on processing video frames captured by the imagesensor.

Optionally, processing video frames comprises applying an optical flowalgorithm to track a direction of movement in the video frames.

Optionally, the image is captured by the image sensor as part of atime-series of video frames.

Optionally, the image is one of a white-light image and a fluorescenceimage.

Optionally, detecting the light emitted by the one or more beacons ofthe display assembly is performed by the image sensor of the handhelddevice.

Optionally, the light emitted by the one or more beacons is infraredlight.

Optionally, the one or more beacons are positioned proximate to aperiphery of the display in respective fixed locations relative to thedisplay.

According to an aspect, a computer program product including computerimplementable instructions, or a non-transitory computer-readablestorage medium for surgical image capture and display is provided, thenon-transitory computer-readable storage medium storing instructions.The instructions are configured to be executed by one or more processorsof a surgical image capture and display system comprising (a) a handheldimage capture and pointing device, the handheld device comprising animage sensor configured to detect white light and infrared light and (b)a display assembly configured to display one or more images captured bythe handheld device, wherein executing the instructions causes thesystem to: capture, by the image sensor of the handheld device, animage; display, by a display of the display assembly, a graphical userinterface comprising the image; while the graphical user interface isdisplayed by the display, detect, by the image sensor, light emitted byone or more beacons of the display assembly; determine, based on thelight emitted by the one or more beacons of the display assembly, aposition or orientation of the handheld device relative to the displayassembly; and update display of the graphical user interface comprisingthe image in accordance with the position or orientation of the handhelddevice.

According to an aspect, a surgical image capture and display method isprovided, the method performed at a surgical image capture and displaysystem comprising (a) a handheld image capture and pointing device, thehandheld device comprising an image sensor configured to detect whitelight and infrared light, (b) a display assembly configured to displayone or more images captured by the handheld device, and (c) one or moreprocessors, the method comprising: capturing, by the image sensor of thehandheld device, an image; displaying, by a display of the displayassembly, a graphical user interface comprising the image; while thegraphical user interface is displayed by the display, detecting, by theimage sensor, light emitted by one or more beacons of the displayassembly; determining, based on the light emitted by the one or morebeacons of the display assembly, a position or orientation of thehandheld device relative to the display assembly; and updating displayof the graphical user interface comprising the image in accordance withthe position or orientation of the handheld device.

According to an aspect, a medical video analysis system, comprising: animage capture device configured to detect white light and infraredlight; a display assembly configured to display one or more imagescaptured by the handheld device; and one or more processors configuredto cause the system to: capture a first series of video frames of targettissue in a first spectrum; capture a second series of video frames ofthe target tissue in a second spectrum, wherein the second series ofvideo frames is captured simultaneously with capturing of the firstseries of video frames; identify a first region of interest in one ormore frames of the first series of video frames, the first region ofinterest corresponding to a first feature of the target tissue apply atracking algorithm to track the first feature of the target tissue inthe first series of video frames, wherein the tracking algorithm isconfigured to track the first feature of the target tissue independentlyfrom tracking of surrounding tissue; and identify a second region ofinterest in one or more frames of the second series of video framesbased on the tracking algorithm applied to the first series of videoframes, the second region of interest corresponding to the first featureof the target tissue.

Optionally, the one or more processors are configured to cause thesystem to apply an image analysis algorithm to the second region ofinterest in the one or more frames of the second series of video frames.

Optionally, the one or more processors are configured to cause thesystem to visualize the second region of interest.

Optionally: the first and second series of video frames capture a firstperiod of time during which a fluorescence dye is not present in thetarget tissue and a second period of time during which the fluorescencedye is present in the target tissue; the tracking algorithm is appliedto a first set of frames, corresponding to both the first period of timeand the second period of time, from the first series of video frames;and the one or more frames of the second series of video frames to whichan image analysis algorithm are applied are corresponding to the secondperiod of time.

Optionally, the one or more frames of the first series of video framesin which the first region of interest is identified correspond to thefirst period of time.

Optionally, the one or more processors are configured to cause thesystem to: identify a third region of interest in one or more frames ofthe first series of video frames, the third region of interestcorresponding to a second feature of the target tissue; apply thetracking algorithm to track the second feature of the target tissue inthe first series of video frames, wherein the tracking algorithm isconfigured to track the second feature of the target tissueindependently from tracking of the surrounding tissue; and identify afourth region of interest in one or more frames of the second series ofvideo frames based on the tracking algorithm applied to the first seriesof video frames, the fourth region of interest corresponding to thesecond feature of the target tissue.

Optionally, the image capture device comprises a first image sensorconfigured to capture the video frames of the first spectrum and asecond image sensor configured to capture the video frames of the secondspectrum.

Optionally, the image capture device comprises a first image sensorconfigured to capture the video frames of the first spectrum and thevideo frames of the second spectrum.

Optionally, identifying the first region of interest corresponding tothe first feature of the target tissue comprises identifying the firstfeature of the target tissue by image analysis of one or more frames ofthe first series of video frames.

Optionally, identifying the first region of interest corresponding tothe first feature of the target tissue comprises receiving a user inputspecifying the first region of interest.

Optionally, receiving the user input specifying the first region ofinterest comprises determining a location on a display at which theimage capture device is aimed, wherein the location on the display isdisplaying the first region of interest at a time at which the imagecapture device is aimed at the location on the display.

Optionally, the first spectrum is a visible light spectrum.

Optionally, the second spectrum is a NIR infrared spectrum.

According to an aspect, a computer program product including computerimplementable instructions, or a non-transitory computer-readablestorage medium for medical video analysis is provided, thenon-transitory computer-readable storage medium storing instructions.The instructions are configured to be executed by one or more processorsof a medical video analysis system comprising an image capture deviceconfigured to detect white light and infrared light and a displayassembly configured to display one or more images captured by thehandheld device, wherein executing the instructions causes the systemto: capture a first series of video frames of target tissue in a firstspectrum; capture a second series of video frames of the target tissuein a second spectrum, wherein the second series of video frames iscaptured simultaneously with capturing of the first series of videoframes; identify a first region of interest in one or more frames of thefirst series of video frames, the first region of interest correspondingto a first feature of the target tissue; apply a tracking algorithm totrack the first feature of the target tissue in the first series ofvideo frames, wherein the tracking algorithm is configured to track thefirst feature of the target tissue independently from tracking ofsurrounding tissue; and identify a second region of interest in one ormore frames of the second series of video frames based on the trackingalgorithm applied to the first series of video frames, the second regionof interest corresponding to the first feature of the target tissue.

According to an aspect, a medical video analysis method is provided, themethod performed at a medical video analysis system comprising an imagecapture device configured to detect white light and infrared light, adisplay assembly configured to display one or more images captured bythe handheld device, and one or more processors, the method comprising:capturing a first series of video frames of target tissue in a firstspectrum; capturing a second series of video frames of the target tissuein a second spectrum, wherein the second series of video frames iscaptured simultaneously with capturing of the first series of videoframes; identifying a first region of interest in one or more frames ofthe first series of video frames, the first region of interestcorresponding to a first feature of the target tissue; applying atracking algorithm to track the first feature of the target tissue inthe first series of video frames, wherein the tracking algorithm isconfigured to track the first feature of the target tissue independentlyfrom tracking of surrounding tissue; and identifying a second region ofinterest in one or more frames of the second series of video framesbased on the tracking algorithm applied to the first series of videoframes, the second region of interest corresponding to the first featureof the target tissue.

According to an aspect, an image capture and display system for tissuesample tracking, comprising: a handheld image capture and pointingdevice; a display assembly configured to display one or more imagescaptured by the handheld device; and one or more processors configuredto cause the system to: capture, by the image capture and pointingdevice, an image; display, by a display of the display assembly, theimage; while the image is displayed by the display, determine a locationon the display assembly at which the handheld device is aimed, whereinthe location on the display corresponds to a region in the displayedimage, wherein the region in the displayed image comprises a targettissue area; and in response to detecting a user input and determiningthe location on the display assembly, generate and store annotation dataassociated with the target tissue area.

Optionally, the annotation data comprises data indicating a region ofinterest in the image.

Optionally, the annotation data comprises a label for the target tissuearea generated based on the user input.

Optionally, the annotation data comprises time-stamp data generatedbased on a time-stamp of the image displayed on the display assembly.

Optionally, the annotation data comprises a unique identifierautomatically generated by the system.

Optionally, the annotation data is stored in association with a tissuesample taken from the target tissue area.

Optionally, the annotation data is stored in association with pathologyresults data associated with the tissue sample taken from the targettissue area.

Optionally: the image is captured by the image capture and pointingdevice, as part of a first time series of video frames of the targettissue in a first spectrum; and the one or more processors cause thesystem to: capture, by the image capture and pointing device, a secondtime series of video frames of the target tissue in a second spectrum;and in response to detecting a user input and in accordance with thedetermined location on the display assembly, identify a region ofinterest in one or more frames of the second series of video frames andapply an image analysis algorithm to the region of interest in the oneor more frames of the second series of video frames.

According to an aspect, a non-transitory computer-readable storagemedium for image capture and display for tissue sample tracking isprovided, the non-transitory computer-readable storage medium storinginstructions configured to be executed by one or more processors of asystem for image capture and display for tissue sample trackingcomprising a handheld image capture and pointing device and a displayassembly configured to display one or more images captured by thehandheld device, wherein executing the instructions causes the systemto: capture, by the image capture and pointing device, an image;display, by a display of the display assembly, the image; while theimage is displayed by the display, determine a location on the displayassembly at which the handheld device is aimed, wherein the location onthe display corresponds to a region in the displayed image, wherein theregion in the displayed image comprises a target tissue area; and inresponse to detecting a user input and determining the location on thedisplay assembly, generate and store annotation data associated with thetarget tissue area.

According to an aspect, a method for image capture and display fortissue sample tracking is provided, the method performed at a system forimage capture and display for tissue sample tracking comprising ahandheld image capture and pointing device, a display assemblyconfigured to display one or more images captured by the handhelddevice, and one or more processors, the method comprising: capturing, bythe image capture and pointing device, an image; displaying, by adisplay of the display assembly, the image; while the image is displayedby the display, determining a location on the display assembly at whichthe handheld device is aimed, wherein the location on the displaycorresponds to a region in the displayed image, wherein the region inthe displayed image comprises a target tissue area; and in response todetecting a user input and determining the location on the displayassembly, generating and storing annotation data associated with thetarget tissue area.

Examples of the methods disclosed herein can be used for non-invasiveimaging of tissue of the patient. The imaged tissue can be externaltissue of the patient, such as skin. Alternatively, the imaged tissuecan be tissue underneath the skin imaged through the intact skin.Alternatively, the imaged tissue can be pre-exposed tissue inside thepatient. Some examples of the methods disclosed herein do not encompassthe step of exposing internal tissue of the patient.

It will be appreciated that any of the aspects, features and optionsdescribed in view of the system(s) apply equally to the method(s),computer program product and computer-readable storage medium(s), andvice versa. It will also be clear that any one or more of the aboveaspects, features and options can be combined. According to an aspect,any one or more of the characteristics of any one or more of thesystems, methods, and/or computer-readable storage mediums recited abovemay be combined, in whole or in part, with one another and/or with anyother features or characteristics described elsewhere herein.

BRIEF DESCRIPTION OF THE FIGURES

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Features will become apparent to those of ordinary skill in the art bydescribing in detail exemplary aspects with reference to the attacheddrawings in which:

FIG. 1 depicts a system for medical image medical image capture,display, manipulation, annotation, and analysis, in accordance with someaspects.

FIG. 2 depicts a view frustum of an image capture device, in accordancewith some aspects.

FIGS. 3A and 3B depict configurations of a camera sensor and display ofa display assembly, in accordance with some aspects.

FIGS. 4A and 4B depict configurations of a camera sensor, in accordancewith some aspects.

FIG. 5 depicts a diagram of a view frustum of an image capture deviceand a calculation of distance based thereon, in accordance with someaspects.

FIGS. 6A and 6B depicts two view frustums of an image capture device inwhich the image capture device is horizontally offset from a center ofthe display assembly, in accordance with some aspects.

FIGS. 7A and 7B depicts a calculation of distances from an image capturedevice to beacons of a display assembly in an arrangement in which theimage capture device is horizontally offset from a center of the displayassembly, in accordance with some aspects.

FIG. 8 depicts a calculation of distance, in accordance with someaspects.

FIG. 9 depicts a calculation of midpoint distance, in accordance withsome aspects.

FIG. 10 depicts a method for capturing a medical image and displaying agraphical user interface comprising the captured image, in accordancewith some aspects.

FIGS. 11A and 11B depict annotated medical images in white light andinfrared light, respectively, in accordance with some aspects.

FIG. 12 depicts a method of capturing medical video and identifyingregions of interest therein, in accordance with some aspects.

FIG. 13 depicts a method of capturing medical images and annotating saidmedical images using a handheld medical image capture and pointingdevice, in accordance with some aspects.

FIG. 14 depicts a computer, in accordance with some aspects.

DETAILED DESCRIPTION

Reference will now be made in detail to implementations and aspects ofvarious aspects and variations of systems and methods described herein.Although several exemplary variations of the systems and methods aredescribed herein, other variations of the systems and methods mayinclude aspects of the systems and methods described herein combined inany suitable manner having combinations of all or some of the aspectsdescribed.

As explained in detail below, this disclosure sets forth systems,methods, and techniques for medical image/video capture, medical imagedisplay, medical image control and manipulation, medical image/videoannotation, and/or feature tracking in medical images/videos.

As explained below, the techniques set out herein may enable capturing,displaying, and manipulating medical images and videos. A handheld imagecapture and pointing device may include one or more image sensorsconfigured to capture both white light video/images and infraredvideo/images. (As used herein, “infrared” or “IR” may refer to infraredand/or near-infrared (“NIR”) light.) The handheld image capture andpointing device may be aimed at a region of tissue of a patient and theone or more image sensors may capture a series of video frames in whitelight and a series of video frames in infrared light. An image from oneof the captured video frames may be displayed on a display of a displayassembly, wherein the display assembly comprises one or more infraredbeacons positioned proximate to the display assembly and configured toemit infrared light. After capturing the image with the handheld device,and while the image is displayed on the display assembly, the operatormay aim the handheld device at the display assembly such that theinfrared beacons are within the field of vision of the handheld deviceand such that the infrared light emitted by the beacons is detected byone or more of the image sensors of the device. Based on the infraredlight detected by the device, a processor associated with the system maydetermine a position and/or orientation of the handheld device relativeto the display assembly, and the system may cause a graphical userinterface including the displayed image to be updated in accordance withthe determined position and/or orientation. For example, the graphicaluser interface may be updated by annotating the displayed image, zoomingthe displayed image or scaling the size of an annotation on the image,rotating the displayed image or an annotation on the image, or the like.Thus, the user of the handheld image capture and pointing device may beable to use a single device both to capture medical images and tomanipulate the images while they are displayed, obviating the need tofrequently switch between different devices during medical procedures.

As also explained below, the techniques set out herein may enable usingframe-synchronized object tracking to identify regions of medical videostreams for analysis and/or visualization. A system for medical imagecapture, display, manipulation, annotation, and analysis may beconfigured such that two (or more) video streams depicting the sametarget tissue may be simultaneously captured, wherein a first one of thevideo streams captures the target tissue in a first spectrum (e.g.,white light) and a second one of the video streams captures the targettissue in a second spectrum (e.g., IR). An object such as a tissuefeature may be tracked in one video stream (e.g., a white light videostream) to determine a region of the other video stream (e.g., an IRvideo stream) at which image analysis should be performed or from whichdata should be extracted. In this manner, a system may be configured touse frame-synchronized video channels (e.g., video streams) to extractinformation out of a first channel of video (e.g., a white light videochannel) and apply that information to corresponding positions in thesecond channel of video (e.g., an IR video channel).

As also explained below, the techniques set out herein may enablecapturing medical images and generating annotation data to be stored inassociation with a region of interest in the medical image and/or inassociation with a tissue sample extracted from the tissue at an areaindicated by the region of interest in the image. A system may capture amedical image (e.g., a frame from a medical video) using a handheldimage capture and pointing device, display said medical image on adisplay assembly, determine a location in said image displayed on saiddisplay assembly at which the handheld device is pointed, and generateand store annotation data associated with a region of interest definedin the image at the location at which the handheld device is pointed.

While the disclosure herein is set forth using examples regardingcapturing, displaying, manipulating, annotating, and analyzing medicalimages and videos, a person of ordinary skill in the art would recognizethat the disclosure herein may be similarly applied to other fields ofendeavor to capture, display, manipulate, annotate, and analyze imagesand videos that are not medical images or medical videos.

Below, FIG. 1 and the accompanying description describe an exemplarysystem for medical image capture, display, manipulation, annotation, andanalysis; FIGS. 2-11 and the accompanying description describe exemplarytechniques for capturing a medical image using a handheld device andcontrolling a graphical user interface comprising the captured imagebased on a determined position and/or determined orientation of thehandheld device with respect to a display assembly on which thegraphical user interface is displayed; FIG. 12 and the accompanyingdescription describe exemplary techniques for simultaneously capturingmultiple medical image video streams in multiple different spectra andidentifying a region of interest in frames of one of the video streamsbased on a tracking algorithm applied to the other video stream; FIG. 13and the accompanying description describes techniques for capturing amedical image using a handheld device and generating annotation data inthe image based on a determined position and/or determined orientationof the handheld device with respect to a display assembly on which thecaptured image is displayed; and FIG. 14 depicts an exemplary computerthat may be used to execute all or part of any one or more of thetechniques described herein.

Exemplary System

FIG. 1 depicts a system 100 for medical image capture, display,manipulation, annotation, and analysis, in accordance with an example.As shown, system 100 may include image capture device 102, imageprocessing and display engine 106, and display assembly 104. Each ofthese components may be communicatively coupled with one or more of theother components such that they may send and receive electronicinformation via network communication amongst one another. For example,as shown by the dotted lines in FIG. 1 , engine 106 may becommunicatively coupled with each of image capture device 102 anddisplay assembly 104.

In this example, image capture device 102 may be any device comprisingone or more image sensors and configured to capture one or more medicalimages and/or videos. The image capture device 102 may comprise multipleimage sensors each configured to capture images/video in differentspectra, for example a white light image sensor and an infrared imagesensor disposed proximate to one another and oriented in essentially thesame direction so as to be able to simultaneously capture images/videoin different spectra of the same tissue area. Image capture device 102may comprise a single image sensor configured to detect light across aplurality of different spectra, and one or more processors associatedwith image capture device 102 may be configured to process signals fromthe single image sensor to generate captured images and/or video inmultiple different spectra based on the light captured by the singleimage sensor.

In this example, image capture device 102 is a handheld image capturedevice configured to be held in the hand of a medical practitioner suchas a surgeon during use. Device 102 may be sufficiently small andlightweight to be able to be held and moved and aimed in one hand by auser. A medical practitioner using device 102 may be able to aim device102 by hand at a tissue area of a patient in order to capture videoand/or images or the patient's tissue. The device 102 may comprise oneor more buttons, triggers, or other input devices that may be activatedby a user in order to cause device 102 to capture a still image and/orto start or stop recording video.

Here, image capture device 102 is communicatively coupled to one or moreother devices and/or systems, such as other components of system 100including image processing and display engine 106, by any wired orwireless electronic communication medium, including by any suitablenetwork communication protocol. Image capture device 102 may beconfigured to transmit captured image data (e.g., medical/surgical imagedata and/or medical/surgical video data), associated metadata, and/orother data to image processing and display engine 106.

In this example, display assembly 104 is a device configured to displayone or more images and/or video captured by image capture device 102 ona display 104 a of display assembly 104. Here, display assembly 104 maybe communicatively coupled to one or more other devices and/or systems,such as other components of system 100 including image processing anddisplay engine 106, by any wired or wireless electronic communicationmedium, including by any suitable network communication protocol.Display assembly 104 may be configured to receive image data (e.g.,medical/surgical image data and/or medical/surgical video data),associated metadata, and/or other data from image processing and displayengine 106 and use said received data to render one or more GUIsincluding captured images/video on display 104 a or display assembly104.

In addition to display 104 a, display assembly 104 may comprise beacons104 b, which may include one or more beacons positioned proximate todisplay 104 a and configured to allow system 100 to determine a positionand/or orientation of device 102 with respect to display assembly 104,as discussed in detail hereinbelow. Beacons 104 b may comprise one ormore fixed objects configured to be visually detectible by an imagesensor of device 102 to be used to allow system 100 to determine aposition and/or orientation of device 102 with respect to displayassembly 104. Beacons 104 b may comprise physical objects visible inwhite light, infrared light, or one or more other spectra; virtualbeacons displayed on display 104 a; and/or dedicated light-emittingelements positioned proximate to display 104 and configured to emitlight visible in white light, infrared light, or one or more otherspectra.

Image processing and display engine 106 may be any device or systemcomprising one or more computer processors configured (1) to receiveimage data, associated metadata, and/or other data from image capturedevice 102, (2) to send processed image data and/or other data directlyor indirectly to display assembly 104, and (3) to perform imageprocessing and GUI control operations as explained in detail hereinbelow. Engine 106 may be configured to generate, update, and/orotherwise provide a GUI for display on display assembly 104, includingdetermining a position and/or orientation of device 102 with respect todisplay assembly 104 and generating, updating, and/or otherwiseproviding the GUI (including captured images controlled therein) basedat least in part on the determined position and/or orientation.

In this example, image capture device 102 may be an image capture andpointing device configured to both capture images/video as describedabove and to serve as a pointing and control device for pointing at andexecuting controls against a graphical user interface (GUI) displayed ona display assembly such as display assembly 104. Exemplary techniques bywhich device 102 may be used to point at and control a GUI displayed ondisplay assembly 104 are described below throughout this disclosure indetail. Device 102 may be used to control a GUI displayed on displayassembly 104 and/or to annotate or otherwise manipulate an imagedisplayed as part of a GUI on display assembly 104 wherein the imagedisplayed as part of the GUI is an image that was previously captured bydevice 102.

In this example, device 102 may be configured to operate selectably ineither an image-capture mode for capturing images/video or in apointer/control mode for pointing at a displayed GUI and/or executingone or more controls against the GUI. Device 102 may be toggled betweenthe two modes based on manual selection by a user (e.g., by a userpressing a button, toggling a switch, or otherwise executing an input).For example, device 102 may be configured to automatically togglebetween the two modes, for example based on an automatic detection bythe device of whether it is aimed at patient tissue and/or whether it isaimed at a display assembly and GUI. For example, system 100 mayautomatically determine whether or not device 102 is aimed at a displayassembly and GUI using automated image analysis and/or trackingalgorithms to analyze images captured by the device; alternatively, oradditionally, system 100 may automatically determine whether or notdevice 102 is aimed at a display assembly and GUI based on detection byone or more sensors of device 102 of light emitted from one or morebeacons of a display assembly, as discussed in further detail below. Inthis example, system 100 may automatically determine whether or notdevice 102 is aimed at a display assembly and GUI based on informationcollected from one or more of a compass, accelerometer, and gyroscopeon-board device 102.

In this example, for example when device 102 is operating inpointer/control mode, control of a GUI and/or an image displayed thereinby device 102 may be based at least in part on a determined orientationof device 102 with respect to display assembly 104 and/or on adetermined position of device 102 with respect to display assembly 104.In this manner, device 102 may be used, for example, as a pointer tocontrol movement of a cursor on a GUI displayed on display assembly 104.Various functionalities for controlling a GUI displayed on display 104based at least in part on determining a location on display assembly 104at which device 102 is aimed may include, for example, displaying acursor, selecting a displayed graphical user interface object,highlighting a displayed graphical user interface object, selecting adisplayed graphical user interface object, rotating a displayedgraphical user interface object, zooming/resizing a displayed graphicaluser interface object, placing a graphical user interface object, and/orgenerating and placing an annotation on an image displayed as part ofthe GUI.

Techniques for determining a location of display assembly 104 at whichdevice 102 is aimed, thereby enabling control of a GUI displayed onassembly 104 by device 102, are explained in detail below with referenceto FIGS. 2-10 .

Exemplary Image Capture, Display, and Manipulation Techniques

FIGS. 3A & 3B show a location on a display corresponding to thecalculated midpoint location and the respective bounding framedimensions. FIG. 3A depicts an orientation in which the sensor of thehandheld device is level with the horizontally-spaced beacons, whereasFIG. 3B depicts an orientation in which the sensor is rotated diagonallywith respect to the beacons. In this example, the calculated boundingframe may dynamically change in size as the sensor is rotated, such thatthe bounding frame may be closer to the edge of the sensor space whenthe sensor is rotated diagonally (e.g., FIG. 3B) as compared to when thesensor is held on a level horizontal with the beacons (e.g., FIG. 3A).As shown, when the camera is rotated then the midpoint may be slightlycloser to the edge of the sensor than if the camera were horizontal.

In this example, the sensor-space inside the dynamically-calculatedbounding frame may be mapped to the display space of the physicaldisplay. The system may be configured such that the entire sensor spacecannot be used, because using the periphery of the sensor space outsidethe calculated bounding frame would cause the system to lose track ofone of the beacons. The system cannot track the midpoint in the sensorspace outside of the bounding frame. If the point that is being trackedis on the edge of the calculated bounding frame (e.g., is on thebounding line), then the corresponding position on the physical display(e.g., the calculated cursor position) may be on a corresponding edge ofthe display space.

The midpoint position in relation to the bounding frame dimensions maybe used to calculate a location on the display at which the camera isaimed (e.g., which may be used to determine a position at which a cursormay be rendered on the GUI). For example, for a surgical display thathas a resolution of 4K UHD (3840×2160 pixels), then the variablesscreenWidth and screenHeight may represent these screen pixeldimensions. A proportion may be used to calculate the position of thecursor: the position of midpointX within the boundingFrameWidth may beproportional to the location cursorX within the screenWidth; and theposition of midpointY within the boundingFrameHeight may be proportionalto the location cursorY within the screenHeight. Cursor position maythus be calculated as follows:

cursorX=(screenWidth*midpointX)/boundingFrameWidth   (1)

cursorY=(screenHeight*midpointY)/boundingFrameHeight   (2)

Beyond calculating a location on the display of a display assembly atwhich the image capture device is aimed (and therefore at which a cursormay be displayed), the system (e.g., system 100) may be configured tocalculate one or more additional properties related to the positionand/or orientation of the handheld image capture device relative to thedisplay assembly.

In this example, the system may be configured to calculate camerarotation. Camera rotation may be determined, for example, by drawing aline between the left and right beacon positions and calculating anangle Θ that the line is off the horizontal, as shown for example inFIGS. 4A and 4B and as shown in the following equation:

Θ=cos⁻¹(abs(rightCx−leftCx)/sensorDistanceBetweenPoints   (3)

In this example, the system may be configured to calculate an actualdistance of the image capture device to the display. Said actualdistance may be calculated, e.g., as follows.

First, the system may calculate a width of a far clipping plane, whichmay be calculated as the width of the field-of-view at the far clippingplane. This can be calculated based on the fact that(fixedLEDDistance/fieldOfViewWidth) is proportional to(sensorDistanceBetweenBpints/sensorWidth), as shown below:

fieldOfViewWidth=(cameraSensorWidth*fixedLEDDistance)/sensorDistanceBetweenPoints  (4)

Then the system may use the right triangle that is created between thefar clipping plane and the camera to calculate the actual distance usingthe tangent of Θ/2 and solve for the adjacent side of the triangle, asshown in FIG. 5 and by the following equation:

actualDistance=(fieldOfViewWidth/2)/tan(Θ/2)   (5)

The system may be configured to calculate an actual distance of theimage capture device to the display in the event that the image capturedevice is horizontally offset from the center of the screen.

In this example, the system may be configured to automatically determinewhether the image capture device is horizontally offset from the displayassembly, for example by determining whether one of the left and rightIR beacons is larger or smaller on the imaging sensor than the other.Based on whether one or the other of the IR beacons is larger or smalleron the imaging sensor, the system may determine whether the imagecapture device is horizontally offset and may determine, based on whichof the IR beacons is larger or smaller on the imaging sensor, thedirection in which the device is offset.

FIGS. 6A and 6B show two examples of the camera sensor being offset froma center of the display assembly. As shown in FIG. 6A, the camera sensoris horizontally offset in a rightward direction from the center of thedisplay assembly, and the right IR beacon, which is closer to the imagesensor, therefore appears larger on the image sensor than the left IRbeacon. As shown in FIG. 6B, the camera sensor is horizontally offset ina leftward direction from the center of the display assembly, and theleft IR beacon, which is closer to the image sensor, therefore appearslarger on the image sensor than the right IR beacon.

The actual diameter of the IR beacon is fixed (fixedLEDDiameter) and maybe known to the system. Thus, the proportion of the fixed diameter onthe far clipping plane as compared to the diameter on the sensor may beused to calculate the field-of-view width. Once the field-of-view widthis known, then the system can use the tangent of the right triangle tocalculate the actual distance from the camera to the left beacon and theactual distance from the camera to right beacon, as shown in FIGS. 7Aand 7B and as shown by the following equations:

leftFieldOfViewWidth=(sensorWidth*fixedLEDDiameter)/leftSensorDiameter  (6)

leftActualDistance=(fieldOfViewWidth/2)/tan(Θ/2)   (7)

rightFieldOfViewWidth=(sensorWidth*fixedLEDDiameter)/rightSensorDiameter  (8)

rightActualDistance=(fieldOfViewWidth/2)/tan(Θ/ 2)   (9)

Once leftActualDistance and rightActualDistance have been calculated,the system may then use the Law of Cosines (c²=a²+b²−2(a)(b)(cos(C)) tocalculate the unknown angle C, as shown in FIG. 8 and as shown by thefollowing equations:

rightActualDistance²=fixedLEDDistance²+leftActualDistance²−2(fixedLEDDistance) (leftActualDistance) (cos C)   (10)

cosC=(fixedLEDDistance²+leftActualDistance²−rightActualDistance²)/(2(fixedLEDDistance)(leftActualDistance))   (11)

Once the angle C is determined, the system may then apply the law ofcosines again to calculate the actual midpoint distance, as shown inFIG. 9 and as shown by the following equations:

midpointActualDistance²=(fixedLEDDistance/2)²+leftActualDistance²−2(fixedLEDDistance/2)(leftActualDistance) (cos C)   (12)

midpointActualDistance=sqrt((fixedLEDDistance/2)²+leftActualDistance²−2(fixedLEDDistance/2)(leftActualDistance) (cos C))   (13)

Once the angle C and the actual midpoint distance are known, the systemmay then use the law of cosines once again to solve for angle B, whereangle B is the angle that the camera is horizontally offset. Thiscalculation is shown by the following equations:

leftActualDistance²=midpointActualDistance²+(fixedLEDDistance/2)²−2(midpointActualDistance)(fixedLEDDistance/2) (cos B)   (14)

cosB=(midpointActualDistance²+(fixedLEDDistance/2)²−2leftActualDistance²)/(2(midpointActualDistance)(fixedLEDDistance/2))   (15)

B=cos⁻¹((midpointActualDistance²+(fixedLEDDistance/2)²−2leftActualDistance²)/(2(midpointActualDistance)(fixedLEDDistance/2)))   (16)

While the above explains techniques in which two horizontally-offset IRbeacons are used, other beacon arrangements may be used; for example, asingle beacon may be used, three (or more) beacons in a line may beused, three beacons in a triangle may be used, and/or four beacons in arectangle may be used.

While the above explains techniques by which visual detection of one ormore beacons may be used to determine a position and/or orientation ofan image capture device, position and/or orientation of an image capturedevice may also be determined based in whole or in part on datacollected from one or more of an accelerometer, a gyroscope, and acompass on board the image capture device. Additionally oralternatively, position and/or orientation of an image capture devicemay also be determined based in whole or in part on other imageprocessing methods (aside from detection of beacons as explained above),for example by using video tracking and/or applying sparse and/or denseoptical flow tracking algorithms to track movement of an image capturedevice based on video frames captured by the image capture device.

FIG. 10 depicts a flowchart representing an exemplary method 1000 forcapturing a medical image and displaying a graphical user interfacecomprising the captured image, in accordance with some aspects.

As described below in detail, method 1000 may enable an image captureand display system to capture medical images using a handheld imagecapture and pointing device; to display one or more of the capturedimages on a display assembly as part of a GUI; to calculate a positionand/or orientation of the image capture and pointing device relative tothe display assembly; and to then control display of the GUI includingthe captured image based at least in part on the determined relativeposition and/or orientation.

The method 1000 may be carried out, in whole or in part, by one or moreof the components of a system for medical image capture, display,manipulation, annotation, and analysis, such as system 100 describedabove with respect to FIG. 1 . Any one or more of the aspects of method1000 may be combined, in whole or in part, with any one or more of theaspects of FIG. 1 , and/or with any one or more of the systems, methods,devices, and/or techniques described elsewhere herein.

At block 1002, in this example, the system may capture a first imageusing an image sensor of a handheld image capture and pointing device.In the example of system 100, image capture and pointing device 102 maycapture an image of tissue of a patient. The first captured image may bea standalone still image; the first captured image may be a frame of aseries of video frames. The first captured image may be an imagecaptured in one or a plurality of spectra in which the image capturedevice is configured to operate; for example, the captured image may bea white light image.

The first captured image may be processed locally and/or transmitted forremote processing. In the example of FIG. 1 , the first captured imagemay be transmitted (by wired or wireless network communication) to imageprocessing and display engine 106 (where it may be processed and/orcaused to be displayed by display assembly 104).

At block 1004, in this example, the system may display a graphical userinterface comprising the first captured image using a display of adisplay assembly. In the example of system 100, image processing anddisplay engine 106 may receive the first captured image from device 102,may optionally process the received first image, and may generate orotherwise provide a graphical user interface comprising the firstcaptured image for display on display 104 by display assembly 104.

In this example, the graphical user interface may be a surgical displaygraphical user interface configured to allow a surgeon to view raw,processed, and/or annotated surgical images and/or video during asurgical procedure in the surgical environment. As discussed herein, thegraphical user interface may be configured to allow a user to annotateone or more displayed images and/or to perform region-of-interesttracking in one or more video streams based on tissue featuresassociated with said annotations.

In this example, the graphical user interface may be configured to allowa user to execute one or more inputs to select, control, manipulate,zoom, rotate, augment, modify, or otherwise control one or more medicalimages displayed as part of the graphical user interface. The graphicaluser interface may be configured to be able to be controlled by anysuitable input device such as a mouse, keyboard, touch-screen controls,or the like. The graphical user interface may be configured to be ableto be controlled by a handheld pointing device, such as image captureand pointing device 102, configured to generate a cursor at a locationin the graphical user interface corresponding to a displayed location onthe display at which the pointing device is aimed.

At block 1006, in this example, while the graphical user interfacecomprising the first image is displayed by the display assembly, thesystem may detect, using an image sensor of the handheld image captureand pointing device, light emitted by one or more beacons of the displayassembly. In the example of system 100, image capture device 102 maycapture light emitted by one or more of beacons 104 b of displayassembly 104.

In this example, the beacons may be positioned proximate to a peripheryof the display in respective fixed locations relative to the display. Asdiscussed above, the beacons may be configured to emit light in one ormore spectra for detection by one or more image sensors of the imagecapture device 102. Light emitted (and/or reflected) by the one or morebeacons may be captured by one or more of the same image sensors used tocapture the first image as discussed above with respect to block 1002.Alternatively, or additionally, light emitted (and/or reflected) by theone or more beacons may be captured by one or more of the same imagesensors used to capture the second image as discussed below with respectto block 1018. Alternatively, or additionally, light emitted (and/orreflected) by the one or more beacons may be captured by one or moreimage sensors not used to capture either the first image or the secondimage.

In this example, one or more of the beacons may be IR beacons. One ormore beacons that do not emit light but merely reflect light may beused. One or more virtual beacons generated and displayed on the displaydisplaying the graphical user interface comprising the first image maybe used.

Information regarding the detected light emitted from (and/or reflectedby) the one or more beacons may be processed locally by the imagecapture device and/or may be transmitted, for example to an imageprocessing and display engine, for processing remotely.

At block 1008, in this example, the system may determine, based on thelight emitted by the one or more beacons of the display assembly, aposition of the handheld image capture and pointing device relative tothe display assembly. At block 1010 the system may determine, based onthe light emitted by the one or more beacons of the display assembly, anorientation of the handheld image capture and pointing device relativeto the display assembly. In the example of system 100, thedeterminations at block 1008 and/or the determination at block 1010 maybe made by image processing and display engine 106, and results of saidone or more determinations may be used by the image processing anddisplay engine to control one or more aspects of a GUI comprising one ormore captured images displayed on display 104.

In this example, determination of a position of a handheld image captureand pointing device relative to a display assembly and/or determinationof an orientation of a handheld image capture and pointing devicerelative to a display assembly may be performed in accordance with allor part of any one or more of the techniques described above withrespect to FIGS. 1-9 . For example, a system may determine a location atwhich a handheld pointing device is aimed (e.g., a “cursor position”) ina plane of a display assembly, a rotation of a handheld pointing devicerelative to a display assembly, and/or an actual distance of a handheldpointing device relative to a display assembly. As described below, thesystem may be configured such that one or more aspects of the displayedGUI including the first captured image displayed therein may becontrolled based on the determined position and/or orientation.

In this example, for example as described above with respect to FIGS.1-9 , determining a position of a handheld image capture and pointingdevice relative to a display assembly may comprise determining adistance from the display assembly to the handheld device. For exampleas described above with respect to FIGS. 1-9 , determining a position ofa handheld image capture and pointing device relative to a displayassembly may comprise determining an offset angle of the handheld devicewith respect to a plane intersecting the display assembly, for example aplane intersecting the display at a right angle along a line from thetop center of the display assembly to the bottom center of the displayassembly.

In this example, for example as described above with respect to FIGS.1-9 , determining an orientation of a handheld image capture andpointing device relative to a display assembly may comprise determininga location in the plane of the display assembly at which the imagecapture and pointing device is aimed. For example as described abovewith respect to FIGS. 1-9 , determining an orientation of a handheldimage capture and pointing device relative to a display assembly maycomprise determining rotational orientation of the handheld device, forexample a rotational orientation of the device with respect to a linealong which the handheld device is aimed.

In this example, in a similar manner as described above with respect toFIGS. 1-9 , a system may similarly determine a relative position and/orrelative orientation of a handheld image capture and pointing devicewith respect to another component of system 100 or another component ofanother system, whether by using the beacon-based determinationtechniques described above and/or by using one or more of the otherposition/orientation determination techniques (e.g., gyroscope,accelerometer, compass, video processing) described above. Saiddetermined position and/or orientation relative to one or more othercomponents of a system may similarly be used to control one or moreaspects of a GUI comprising one or more captured images displayed on adisplay.

At block 1012, in this example, the system may update display of thegraphical user interface comprising the first image in accordance withthe determined position and/or the determined orientation. In theexample of system 100, engine 106 may update one or more features oraspects of the GUI comprising the first captured image displayed ondisplay 104 a.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise controlling a displayedcursor on the GUI, selecting an image for display on the GUI, zooming animage on the GUI, generating an annotation on an image on the GUI,rotating an image on the GUI, resizing an annotation on an image on theGUI, rotating an annotation on an image on the GUI, and/or selecting orupdating a display size for an object and/or a font size for text on theGUI.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise displaying a cursor onthe graphical user interface at a location on the display at which thehandheld device is determined to be aimed. A user may then press one ormore buttons on the handheld device to execute a cursor “click” in theGUI.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise generating and displayingan annotation on the displayed image at a location on the display atwhich the handheld device is determined to be aimed. For example, a usermay point the handheld device at a location on the display displaying atissue feature and the system may generate an annotation (e.g., adisplayed, numbered, drawn, and/or labeled bounding box) at the locationat which the device is aimed. The user may label one or more regions ofinterest in an image of tissue in this manner, and the labeled regionsof interest may be stored for future review, used for image analysis,and/or used for feature tracking in one or more video streams.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise rotating an imageannotation in accordance with the orientation of the handheld device. Animage (e.g., the first captured image) displayed on the GUI may berotated in accordance with rotation of the handheld device. Anotherfeature of the GUI, such as but not limited to an image annotation(e.g., a region of interest bounding box) may be rotated in accordancewith rotation of the handheld device.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise modifying a size of agraphical user interface object such as a displayed image and/or animage annotation in accordance with the position of the handheld device.For example, a user may be able to push the handheld device toward thedisplay assembly to decrease the size of the GUI object and pull thehandheld device away from the display assembly to increase the size ofthe GUI object. The system may be configured to automatically set thesize of one or more GUI objects (e.g., an image display window, text onthe GUI) in accordance with a determined distance of the handheld devicefrom the display assembly, for example to ensure that text is legible bya user of the handheld device when the user is standing at variousdistances from the display assembly.

In this example, updating display of the graphical user interfacecomprising the first image in accordance with the determined positionand/or the determined orientation may comprise displaying a notificationbased on an offset angle determined in accordance with the determinedposition of the handheld device. For example, the system may beconfigured to display a warning or other notification when handhelddevice is at an offset angle that determines a predefined thresholdoffset angle, such that the user may be made aware that his viewingangle is non-optimal.

At block 1014, in this example, the system may, in accordance with thedetermined position and/or the determined orientation, identify a firstregion of interest in the first image. In the example of system 100,image processing and display engine 106 may identify the first region ofinterest. The identified region of interest may be a region of interestthat is “tagged” by a user of the handheld device when the user pointsthe handheld device at a portion of the image displaying the region tobe tagged and executes an input such as pressing one or more buttons onthe handheld device. The system may generate an annotation associatedwith the tagged region of interest in the image, including by generatingand displaying a visual marker such as a bounding box highlighting theannotated region and/or by generating and storing metadata associatedwith the annotated region, wherein the metadata may include a locationin the image, a time of the annotation, an unique identifier of theannotation, a label of the annotation, time stamp data for theannotation, information indicating a user who created the annotation,and/or information indicating a manner in which the annotation wascreated.

At block 1016, in this example, the system may apply a trackingalgorithm and/or an image analysis algorithm to the first region ofinterest. In the example of system 100, image processing and displayengine 106 may apply said tracking algorithm and/or image analysisalgorithm.

In this example, an image analysis algorithm applied may determine atype of tissue associated with the first region of interest, a health oftissue associated with the first region of interest, a level ofperfusion of tissue associated with the first region of interest, and/orany other medically relevant information that may be discerned from theimage of the tissue associated with the first region of interest.

In this example, a tracking algorithm applied may track one or morefeatures of tissue associated with a region of interest across aplurality of video frames of which the first captured image is a part.Thus, a user may tag a tissue feature in the first captured image andthe system may track (e.g., automatically locate and identify) thetissue feature across one or more other frames in a video stream fromwhich the first captured image is taken.

In this example, the system may be configured to output data and/orimage portions of a portion of tissue in another frame of the videostream wherein the data is based on a portion or portions of the one ormore other frames identified by the tracking algorithm. The system maybe configured to apply one or more image analysis algorithms to aportion or portions of one or more other frames in the video streamidentified by the tracking algorithm.

At block 1018, in this example, the system may capture a second imageusing an image sensor of the handheld image capture and pointing device.In the example of system 100, image capture and pointing device 102 maycapture the second image of tissue of the patient. The second capturedimage may be a standalone still image; alternatively, the secondcaptured image may be a frame of a series of video frames.

In this example, the second captured image may be an image captured inone or a plurality of spectra in which the image capture device isconfigured to operate; for example, the captured image may be an IRimage. The tissue captured in the second image may be the same tissue ascaptured in the first image. For example, the image capture device maybe configured to use a single set of one or more image sensors sensitivein multiple different spectra to capture both the first and secondimages, or the image capture device may be configured to use separatesets of one or more image sensors sensitive in different respectivespectra to capture the first and second images respectively.

In this example, the tissue may be the same tissue captured in the firstimage. The second captured image may be captured simultaneously oressentially simultaneously (e.g., within less than or equal to 1 second,0.1 seconds, 0.01 seconds, or 0.001 seconds) with the first image, suchthat the first and second images may represent the same tissue area atthe same or essentially the same period in time in different spectra.The first and second images may be corresponding frames representing thesame or essentially the same moment in time in simultaneously-capturedvideo streams of the target tissue.

In this example, the second captured image may be processed locallyand/or transmitted for remote processing. In the example of FIG. 1 , thesecond captured image may be transmitted (by wired or wireless networkcommunication) to image processing and display engine 106 (where it maybe processed and/or caused to be displayed by display assembly 104).

At block 1020, in this example, the system may apply a trackingalgorithm and/or image analysis algorithm to a second region ofinterest, wherein the second region of interest is in the second imageand corresponds to the first region of interest. Block 1020 may followfrom both block 1018 and block 1014. In the example of system 100, imageprocessing and display engine 106 may apply said tracking algorithmand/or image analysis algorithm.

In this example, the system may determine the second region of interestin the second image based on the first region of interest in the firstimage. For example, when the first and second image depict the sametissue region at the same point in time, the second region of interestmay be selected to have the same pixel coordinates in the second imageas the pixel coordinates of the first region of interest in the firstimage.

In this example, the system may be configured to select the secondregion of interest in the second image based on spatial correspondencewith the first region of interest (e.g., as indicated by a user usingthe pointing device) in the first image, and to then apply a trackingalgorithm to a video stream from which the second image was taken totrack a tissue feature in multiple frames of the video stream of thesecond image.

In this example, the system may be configured to select the secondregion of interest in the second image based on spatial correspondencewith a region of interest in a video stream from which the first imagewas taken. For example, a user may indicate the first region of interestcorresponding to a tissue feature in the first image, and the system maythen track the tissue feature to a subsequent frame of the first videostream. The system may then correlate the subsequent frame of the firstvideo stream spatially to the second image in the event that thesubsequent frame and the second image correspond to the same point intime.

In another example, instead of spatially correlating the first image orthe subsequent frame in the first video stream to the second imageitself, the system may spatially correlate the first image or thesubsequent frame in the first video stream to a prior frame in thesecond video stream, wherein the prior frame was captured at an earliertime in the second video stream than the second image. The system mayuse the spatial correlation to identify a tissue feature in the secondvideo stream starting at the prior frame and may apply a trackingalgorithm to track the tissue feature in the second video stream toselect the second region of interest (corresponding to the tissuefeature) in the second image.

It should be noted that image tracking algorithms may be applied with orwithout regard for a time-direction of frames from a video stream, suchthat one or more tissue features may be traced forward and/or backwardin one or more video streams.

In this example, the system may apply an image analysis algorithm to thesecond region of interest in the second image once the image analysisalgorithm is identified therein. The image analysis algorithm appliedmay determine a type of tissue associated with the second region ofinterest, a health of tissue associated with the second region ofinterest, a level of perfusion of tissue associated with the secondregion of interest, and/or any other medically relevant information thatmay be discerned from the image of the tissue associated with the secondregion of interest.

It may be advantageous to allow a practitioner to tag a region ofinterest in a first video stream and/or to automatically track a tissuefeature in a first stream because the first stream may be more amenableto human and/or automated visual perception at certain points in time.For example, if a first video stream is a white light stream and asecond video stream is a fluorescence IR stream, the amount of lightcaptured in the second video stream may be very low when there is not a,e.g. pre-administered, bolus of fluorescence agent traveling throughvasculature of the target tissue. Thus, it may be advantageous to tracka tissue feature in a white light video stream before the imaging agentis present in the tissue and to then observe the fluorescence videostream of the feature and/or apply one or more image analysis algorithmsto the fluorescence video stream of the feature only at the time thatthe imaging agent is present in the tissue.

FIGS. 11A and 11B, discussed in further detail below with respect toexemplary tracking techniques and exemplary image annotation techniques,show corresponding annotated images of a target tissue area, wherein theimage in FIG. 11A is captured in white light and the image in FIG. 11Bis captured in infrared spectrum and rendered in grayscale. As shown,the two images have been annotated with corresponding regions ofinterest, wherein each image includes regions of interest numbered 1-5and wherein similarly numbered regions of interest in the imagescorrespond to one another by indicating the same tissue features in bothimages. Thus, for example, region of interest “1” in the image in FIG.11A may be the region of interest in the first captured image in method1000, and region of interest “1” in the image in FIG. 11B may be theregion of interest in the second captured image in method 1000.

Exemplary Tracking Techniques

As described above, a system for medical image capture, display,manipulation, annotation, and analysis, such as system 100, may beconfigured such that two (or more) video streams depicting the sametarget tissue may be simultaneously captured, wherein a first one of thevideo streams captures the target tissue in a first spectrum (e.g.,white light) and a second one of the video streams captures the targettissue in a second spectrum (e.g., IR). In this example, an object suchas a tissue feature may be tracked in one video stream (e.g., a whitelight video stream) to determine a region of the other video stream(e.g., an IR video stream) at which image analysis should be performedor from which data should be extracted. In this manner, a system may beconfigured to use frame-synchronized video channels (e.g., videostreams) to extract information out of a first channel of video (e.g., awhite light video channel) and apply that information to correspondingpositions in the second channel of video (e.g., an IR video channel).

Described below are exemplary techniques for performing tracking inframe-synchronized video streams.

In this example, a system may be configured to track a tissue feature ina white light video stream and to then extract information from oranalyze a corresponding location in a frame-synchronized IR video streamat a time at which a bolus of fluorescence agent is moving through thetissue vasculature and emitting IR fluorescence emission light. Thesystem may track one or more regions of interest in white light videosince white light video may have more information than fluorescencevideo that is advantageous to the performance of the tracking algorithm.As the white light video sequence continues, the tracking algorithm maylock-onto one or more of these regions of interest and track theircoordinates throughout the white light video sequence. The region ofinterest coordinates that are tracked in the white light video channelmay then be used to identify corresponding regions in the IR videochannel and to do analysis on (e.g., extract data from and/or performone or more image analysis techniques on) said corresponding regionsand/or visualize said corresponding regions. Thus, tissue tracking maybe effectively performed in an information-rich white light spectrum andthen fluorescence emission information may be extracted from an IRfluorescence emission spectrum at regions selected by the trackingalgorithm.

One or both video feeds used in performance of the tracking techniquesdescribed herein may be captured using a handheld image capture andpointing device, such as device 102 of system 100.

The image capture device may be configured to use a single set of one ormore image sensors sensitive in multiple different spectra to captureboth the first and second video streams. The image capture device may beconfigured to use separate sets of one or more image sensors sensitivein different respective spectra to capture the first and second videostreams respectively. Both sets of one or more sensors maysimultaneously capture images of the same target tissue.

In this example, after capturing one or both of the video streams, theimage capture device may transmit (e.g., via wired or wireless networkcommunication) the captured video stream(s) to a processing device forapplication of one or more of the tracking techniques described herein.The tracking techniques described herein may be applied by imageprocessing and display engine 106. Engine 106 may be communicativelycoupled to one or more electronic data stores for storing sequences offrames from one or more received video streams. The tracking techniquesdescribed herein may be performed in real time as the video streams arereceived and/or on pre-recorded and stored video streams.

FIGS. 11A and 11B show corresponding annotated images of a target tissuearea, wherein the image in FIG. 11A is captured in white light and theimage in FIG. 11B is captured in infrared spectrum and rendered ingrayscale. As shown, the two images have been annotated withcorresponding regions of interest, wherein each image includes regionsof interest numbered 1-5 and wherein similarly numbered regions ofinterest in the images correspond to one another by indicating the sametissue features in both images. Thus, for example, region of interest“1” in the image in FIG. 11A may be a region of interest that is or hasbeen tracked in a white light video stream, and region of interest “1”in the image in FIG. 11B may be a corresponding region in a frame of theIR video stream, selected on the basis of the tracking algorithm appliedto the white light video stream, on which the system may perform dataextraction or image analysis.

FIG. 12 depicts a flowchart representing an exemplary method 1200 ofcapturing medical video and identifying regions of interest therein, inaccordance with some aspects.

As described below in detail, method 1200 may enable an image captureand display system to capture multiple streams of medical video (e.g.,using a handheld image capture and pointing device); to track one ormore features of target tissue in a first video stream; to identify,based on said tracking, a region depicting the feature of the targettissue in a second video stream; and to apply one or more image analysisalgorithms and/or visualization techniques to the region depicting thefeature of the target tissue in the second video stream.

In this example, method 1200 may be carried out, in whole or in part, byone or more of the components of a system for medical image capture,display, manipulation, annotation, and analysis, such as system 100described above with respect to FIG. 1 . Any one or more of the aspectsof method 1200 may be combined, in whole or in part, with any one ormore of the aspects of FIG. 1 , and/or with any one or more of thesystems, methods, devices, and/or techniques described elsewhere herein.

At block 1202, in this example, the system may capture a first series ofvideo frames of target tissue in a first spectrum. In the example ofsystem 100, image capture and pointing device 102 may capture the firstvideo stream (e.g., a time series of video frames) of tissue of apatient. The first series of video frames may be captured in one or aplurality of spectra in which the image capture device is configured tooperate; for example, the first captured video stream may be a whitelight video stream.

In this example, the first captured video stream may be processedlocally and/or transmitted for remote processing. In the example of FIG.1 , the first captured video stream may be transmitted (by wired orwireless network communication) to image processing and display engine106 (where it may be processed and/or caused to be displayed by displayassembly 104).

At block 1204, in this example, the system may capture a second seriesof video frames of the target tissue in a second spectrum.

In the example of system 100, image capture and pointing device 102 maycapture the second video stream (e.g., a time series of video frames) oftissue of a patient. The second series of video frames may be capturedin one or a plurality of spectra in which the image capture device isconfigured to operate; for example, the second captured video stream maybe an IR video stream.

The second captured video stream may be processed locally and/ortransmitted for remote processing. In the example of FIG. 1 , the secondcaptured video stream may be transmitted (by wired or wireless networkcommunication) to image processing and display engine 106 (where it maybe processed and/or caused to be displayed by display assembly 104).

In a similar manner as described above the image capture device may beconfigured to use a single set of one or more image sensors sensitive inmultiple different spectra to capture both the first and second videostreams. The image capture device may be configured use separate sets ofone or more image sensors sensitive in different respective spectra tocapture the first and second video streams respectively. Both sets ofone or more sensors may face in the same direction and thereforesimultaneously capture video streams of the same target tissue.

In this example, the first and second video streams may be capturedsimultaneously with one another. Corresponding frames of the first andsecond video streams may be captured simultaneously or essentiallysimultaneously (e.g., within less than or equal to 1 second, 0.1seconds, 0.01 seconds, or 0.001 seconds) with one another, such that thecorresponding frame may represent the same tissue area at the same oressentially the same period in time in different spectra. Thecorresponding frames may thus represent the same or essentially the samemoment in time in the simultaneously-captured video streams of thetarget tissue.

At block 1206, in this example, the system may identify a first regionof interest in the first series of video frames, the first region ofinterest corresponding to a first feature of the target tissue. In theexample of system 100, image processing and display engine 106 mayidentify the first region of interest. The first region of interest maybe identified in one or more frames of the first series of video frames.The system may automatically identify the first region of interest(e.g., using one or more image analysis or object recognitionalgorithms).

The system may identify the first region of interest based at least inpart on user input indicating the region in one or more frames of thefirst series of video frames. A user may use a mouse, keyboard,touch-screen, or other input device to indicate a portion of a displayedimage from the first series of video frames, such as by selecting acenter of the region of interest or drawing a bounding box (or otherbounding shape) for the region of interest. A user may specify the firstregion of interest by using a handheld image capture and pointing device(e.g., device 100), for example by using said device to point at aportion of an image (e.g., with a cursor) displayed on a displayassembly (e.g., display assembly 104) to generate a tag or bounding boxor to otherwise cause the system to identify the portion of the image atwhich the user is pointing as the first region of interest. The systemmay select the first region of interest based on a portion of adisplayed image at which a handheld image capture and pointing device isaimed in accordance with all or part of any of the techniques describedabove with respect to FIGS. 2-10 and method 1000, for example blocks1004-1014 of method 1000.

At block 1208, in this example, the system may apply a trackingalgorithm to track the first feature of the target tissue in the firstseries of video frames, wherein the tracking algorithm is configured totrack the first feature independently from tracking of surroundingtissue. In the example of system 100, image processing and displayengine 106 may apply said tracking algorithm. In this example, thetracking algorithm may be configured to track the first feature throughvarious frames (e.g., forward or backward in time) of the first seriesof video frames based on the recognition (e.g., by image analysisperformed separate from and/or as a part of the tracking algorithm) ofthe tissue feature in the first region of interest identified by thesystem in one or more frames of the first series of video frames. Thus,for example, a user may tag the first region of interest containing thetissue feature in one frame of the first video stream, and the systemmay then track the tissue feature throughout the rest of the first videostream.

In this example, the tracking algorithm configured to track the firstfeature independently from tracking of surrounding tissue may be (or mayinclude) a Discriminative Correlation Filter. The DiscriminativeCorrelation Filter may be a Discriminative Correlation Filter withChannel and Spatial Reliability. The algorithm may be a video processing(e.g., computer vision) type of algorithm. The tracking algorithm mayinclude one or more machine learning components.

The system may be configured to track every pixel in a frame using adense optical flow approach. In this approach, the system may track themovement (e.g., motion vectors) of every pixel through the videosequence. The system may track the motion of every pixel withoutconsidering one or more ROI's with respect to motion tracking. A visiblelight series of video frames may still be used to determine thecorresponding positions in a corresponding IR series of video frames.

At block 1210, in this example, the system may identify a second regionof interest in the second series of video frames, based on the trackingalgorithm applied to the first series of video frames, the second regionof interest corresponding to the first region of interest. In theexample of system 100, the identification of the second region ofinterest may be performed by image processing and display engine 106.The system may be configured to select the second region of interest ina frame of the second series of video frames based on spatialcorrespondence with the first region of interest and/or based on spatialcorrespondence to the first tissue feature that may be trackedthroughout the first series of video frames. For example, a user mayindicate the first region of interest corresponding to the tissuefeature in one or more frames of the first series of video frames, andthe system may then track the tissue feature to a different frame of thefirst series of video frames. The system may then correlate thedifferent frame from the first series of video frames spatially to aframe from the second series of video frames.

Spatial correspondence may be used between time-synchronizedframes—e.g., frames representing the same or essentially the same momentin time—across the different series of video frames. Spatialcorrespondence of regions of interest may refer to regions of interestin time-synchronized frames that have the same or similar pixelcoordinates in the frames. For example, in the example of FIGS. 11A and11B, the pair of regions of interest labeled “1” in the respectiveimages have a spatial correspondence with one another, as do the pairslabeled “2,” “3,” “4,” and “5.” Thus, as used herein in describingmethod 1200, it should be understood that the “first region of interest”and the “second region of interest” may depict a similar or identicaltissue area in different images.

Tracking of a tissue feature in a region of interest may be performedbefore and/or after assessing one or more spatial correspondences toidentify the second region of interest in a frame of the second videostream. For example, a user may indicate the first region of interestcontaining the tissue feature in a first frame of the first videostream. The system may determine a spatially corresponding second regionof interest in a first time-synchronized frame of the second videostream (that is time-synchronized with the first frame in the firststream). The system may first track the tissue feature to a second videoframe in the first video stream, and the system may then determine thespatially corresponding second region of interest in a secondtime-synchronized frame of the second video stream (that istime-synchronized with the second frame in the first stream). The systemmay then optionally perform additional tracking on the tissue feature inone or both of the first and second streams after the identification ofthe second region of interest.

At block 1212, in this example, the system may apply an image analysisalgorithm to the second region of interest. At block 1214, in thisexample, the system may visualize the second region of interest. One orboth of block 1212 and block 1214 may be performed following block 1210and the identification of the second region of interest in one or moreframes of the second series of video frames. In the example of system100, image analysis may be performed by image processing and displayengine 106 and visualization may be performed by image processing anddisplay engine 106 causing display 104 a of display assembly 104 todisplay the second region of interest.

In this example, the system may be configured to output data and/orimage portions of the second region of interest. The system may beconfigured to apply one or more image analysis algorithms to the secondregion of interest. An image analysis algorithm applied may determine atype of tissue associated with the second region of interest, a healthof tissue associated with the second region of interest, a level ofperfusion of tissue associated with the second region of interest,and/or any other medically relevant information that may be discernedfrom the second region of interest.

In method 1200, the system may be configured such that image trackingmay be performed in white light at a time before fluorescence agent ispresent in the target tissue, and then image analysis or visualizationor information extraction may be performed in IR light at a time when afluorescence agent emitting IR agent is present in the vasculature ofthe target tissue.

For example, the first and second series of video frames may be capturedduring a first period of time during which a fluorescence dye is notpresent in the target tissue and a second period of time during whichthe fluorescence dye is present in the target tissue; the trackingalgorithm may be applied to a first set of frames, corresponding to boththe first period of time and the second period of time, from the firstseries of video frames; and the one or more frames of the second seriesof video frames to which an image analysis algorithm is applied maycorrespond to the second period of time. The one or more frames of thefirst series of video frames in which the first region of interest isidentified may correspond to the first period of time.

In this example, for example as shown in FIGS. 11A and 11B, the systemmay be configured to simultaneously track multiple different regions ofinterest and/or multiple different tissue features in a single videostream (and/or in a single set of frame-synchronized video streams). Twoor more of the tracked regions or tracked features may be trackedindependently of surrounding tissue.

Exemplary Image Annotation Techniques

As described above, a system for medical image capture, display,manipulation, annotation, and analysis, such as system 100, may beconfigured to capture a medical image and to generate annotation data tobe stored in association with a region of interest in the medical imageand/or in association with a tissue sample extracted from the tissue atan area indicated by the region of interest in the image. The medicalimage can be obtained by an operator of the system before, during, orafter a medical practitioner extracts the tissue sample. The system maybe configured to capture a medical image and to generate annotation datato be stored in association with a region of interest in the medicalimage and/or in association with a tissue sample that has beenextracted, or is about to be extracted, from the tissue at an areaindicated by the region of interest in the image. A system may capture amedical image (e.g., a frame from a medical video) using a handheldimage capture and pointing device, display the captured medical image ona display assembly, determine a location in said image displayed on saiddisplay assembly at which the handheld device is pointed, and generateand store annotation data associated with a region of interest definedin the image at the location at which the handheld device is pointed.

During a medical/surgical procedure, a surgeon or other practitioner maycapture an image from a video stream (e.g., a live video stream beingcaptured by a handheld image capture device such as device 102) shortlybefore the practitioner intends to remove a tissue sample for analysis.Once the image is captured, the practitioner may then mark one or moreregions of interest in that image such that the regions of interest andany associated metadata may be associated with the tissue sample to betaken. The captured image and data associated with the region ofinterest may be stored in association with the tissue sample to beremoved. Alternatively, or additionally, the practitioner may thenremove the tissue sample from the anatomy and the captured image anddata associated with the region of interest may be stored in associationwith the removed tissue sample and/or in association with any datagenerated from analysis of the removed tissue sample.

Examples of images annotated with regions of interest are shown in FIGS.11A and 11B, with FIG. 11A showing a white light image and FIG. 11Bshowing an IR image rendered in grayscale, with both images having fiveregions of interest marked thereon.

In this example, one or more regions of interest may be marked inresponse to a user executing a user input indicating a portion of theimage at which the region of interest should be marked.

In this example, the system may identify the region of interest based atleast in part on user input indicating the region in the image using amouse, keyboard, touch-screen, or other input device to indicate aportion of the displayed image, such as by selecting a center of theregion of interest or drawing a bounding box (or other bounding shape)for the region of interest. A user may specify the region of interest byusing a handheld image capture and pointing device (e.g., device 100),for example by using said device to point at a portion of an image(e.g., with a cursor) displayed on a display assembly (e.g., displayassembly 104) to generate a tag or bounding box or to otherwise causethe system to identify the portion of the image at which the user ispointing as the region of interest. The system may select the firstregion of interest based on a portion of a displayed image at which ahandheld image capture and pointing device is aimed in accordance withall or part of any of the techniques described above with respect toFIGS. 2-10 and method 1000, for example blocks 1004-1014 of method 1000.The identified region of interest may be a region of interest that is“tagged” by a user of the handheld device when the user points thehandheld device at a portion of the image displaying the region to betagged and executes an input such as pressing one or more buttons on thehandheld device.

The system may generate an annotation associated with the tagged regionof interest in the image, including by generating and displaying avisual marker such as a bounding box highlighting the annotated regionand/or by generating and storing metadata associated with the annotatedregion, wherein the metadata may include a location in the image, a timeof the annotation, an unique identifier of the annotation, a label ofthe annotation, time stamp data for the annotation, informationindicating a user who created the annotation, and/or informationindicating a manner in which the annotation was created.

The system may be configured to accept one or more inputs from a user ofthe system to generate and store metadata associated with the region ofinterest. One or more of these inputs may be input to the system using amicrophone (e.g., for voice input), mouse, keyboard, touch-screen, orthe like. One or more of these inputs may be input to the system using ahandheld image capture and pointing device (e.g., device 100), includingby using one or more buttons, keys, microphones, touch-screen devices,or the like mounted thereon. One or more of these inputs may be input tothe system using a handheld image capture and pointing device (e.g.,device 100), including by generating metadata based at least in part ona determined location and/or orientation of the handheld device (e.g.,relative to a display assembly) at the time the annotation and/ormetadata are created.

Time stamp metadata associated with an annotation (e.g., region ofinterest) may include time-stamp data indicating a time at which theimage (e.g., video frame) was captured and/or time-stamp data indicatinga time at which the annotation was created or updated.

In this example, the system may be configured to automatically generatea unique identifier for each of a plurality of regions of interestgenerated by the system. The system may automatically increment a uniqueidentifier each time the system creates a new annotation. Uniqueidentifiers may be unique across a surgical procedure, a set of surgicalprocedures, a patient's medical history, a medical practitioner'sprofessional history, a hospital, a surgical system or platform, and/ora laboratory.

Tissue samples before being sent to pathology may be labeled with thelabels from the region of interest where the sample was taken. Once thepathology results are received, the same label may be used to correlatethe pathology results to the regions of interest as well as the image ofthe anatomy and the time stamp in the video from which the image wastaken.

The region of interest coordinates may then be tracked in the whitelight video channel and may then be used to analyze and/or visualizecorresponding regions in the IR video channel, for example as discussedabove with respect to method 1200.

FIG. 13 depicts a flowchart representing an exemplary method 1200 ofannotating medical images, in accordance with some aspects.

As described below in detail, method 1300 may enable an image captureand display system to capture a medical image (e.g., a frame from amedical video) using a handheld image capture and pointing device; todisplay said medical image on a display assembly; to determine alocation in said image displayed on said display assembly at which thehandheld device is pointed; and to generate and store annotation dataassociated with a region of interest defined in the image at thelocation at which the handheld device is pointed.

In this example, method 1300 may be carried out, in whole or in part, byone or more of the components of a system for medical image capture,display, manipulation, annotation, and analysis, such as system 100described above with respect to FIG. 1 . Any one or more of the aspectsof method 1300 may be combined, in whole or in part, with any one ormore of the aspects of FIG. 1 , and/or with any one or more of thesystems, methods, devices, and/or techniques described elsewhere herein.

At block 1302, in this example, the system may capture a first image ina first spectrum using a handheld image capture and pointing device. Inthe example of system 100, image capture and pointing device 102 maycapture an image of tissue of a patient. The first captured image may bea standalone still image; alternatively, the first captured image may bea frame of a series of video frames. The first captured image may be animage captured in one or a plurality of spectra in which the imagecapture device is configured to operate; for example, the captured imagemay be a white light image.

The first captured image may be processed locally and/or transmitted forremote processing. In the example of FIG. 1 , the first captured imagemay be transmitted (by wired or wireless network communication) to imageprocessing and display engine 106 (where it may be processed and/orcaused to be displayed by display assembly 104).

In this example, capturing the image at block 1302 may share any one ormore characteristics in common with capturing an image as describedabove with reference to block 1002 of method 1000.

At block 1304, in this example, the system may display the first imageusing a display of a display assembly. In the example of system 100, theimage may be displayed by image processing and display engine 106causing display 104 a of display assembly 104 to display the capturedimage. Displaying the image at block 1304 may share any one or morecharacteristics in common with displaying a captured image as describedabove with reference to block 1004 of method 1000.

At block 1306, in this example, while the first image is displayed onthe display of the display assembly, the system may determine a locationon the display assembly at which the handheld image capture device isaimed, wherein the location on the display assembly corresponds to aregion in the first image, wherein the region in the first imagecomprises a target tissue area. In the example of system 100, saiddetermination may be made by image processing and display engine 106.The system may make said determination based on one or more sensorsonboard the handheld device (e.g., gyroscope, accelerometer, compass),based on video processing of video captured by the handheld device,and/or based on optically detecting and analyzing light emitted by oneor more beacons of the display assembly on which the image is displayed.Making said determination may share any one or more characteristics incommon with determining a location at which a handheld device is aimedas described above with reference to FIGS. 2-9 and blocks 1006-1010 ofmethod 1000.

At block 1308, in this example, the system may, in response to detectinga user input and determining the location on the display assembly,generate and store annotation data associated with the target tissuearea. In the example of system 100, image processing and display engine106 may detect a user input, such as a button on handheld device 102 orelsewhere being pressed, and may responsively generate and storeannotation data associated with the target tissue area. As discussedabove, the annotation data may include a bounding box or otherindication of the portion of the image corresponding to the region ofinterest and may further include metadata regarding the tissue area,procedure, patient, practitioner, time stamp data, label data, and/orunique identifier data. Said annotation data may be stored inassociation with and/or as a part of the annotated image and/or a videoof which the annotated image is a frame.

Blocks 1310 and 1318 describe how method 1300 may includeframe-synchronized tracking techniques similar to those described abovewith respect to method 1200. For example, a tissue feature associatedwith a region of interest that is tagged for tissue tracking by asurgeon may be tracked in video and/or analyzed/visualized in acorresponding video in a different spectrum using the same or similartechniques as described above with respect to method 1200.

At block 1310, in this example, the system may capture a second image ina second spectrum using the handheld image capture and pointing device.In the example of system 100, image capture and pointing device 102 maycapture the second image of tissue of the patient. The second image maybe an image captured in a different spectrum than the first image; forexample, the second image may be an IR image. Capturing the second imageat block 1310 may share any one or more characteristics in common withcapturing a captured image as described above with reference to block1018 of method 1000.

At block 1312, in this example, the system may, in response to detectinga user input and determining the location on the display assembly,identify a region of interest in one or more frames of a series of videoframes of which the second image is a part, and apply an image analysisalgorithm, data extraction operation, and/or visualization operation tothe region of interest in the one or more frames of the series of videoframes of which the second image is a part. In the example of system100, said user input detection, region identification, image analysis,data extraction, and/or visualization may be performed by imageprocessing and display engine 106. A surgeon or other practitioner mayaccordingly tag a region of interest in a frame of a white light video,and the system may then track a tissue feature in the region of interestin the white light video and determine a spatially corresponding regionin one or more frames of a frame-synchronized IR video forvisualization, data extraction, and/or analysis of the IR video.

Exemplary Computer

FIG. 14 illustrates a computer, in accordance with some aspects.Computer 1400 can be a component of a system for medical image capture,display, manipulation, annotation, and/or analysis, such as system 100and/or any of its subcomponents described above with respect to FIG. 1 .Computer 1400 may be configured to execute a method for medical imagecapture, display, manipulation, annotation, and/or analysis, such as allor part of any one or more of methods 1000, 1200, and 1300 describedabove with respect to FIGS. 10, 12, and 13 .

Computer 1400 can be a host computer connected to a network. Computer1400 can be a client computer or a server. As shown in FIG. 14 ,computer 1400 can be any suitable type of microprocessor-based device,such as a personal computer; workstation; server; or handheld computingdevice, such as a phone or tablet. The computer can include, forexample, one or more of processor 1410, input device 1420, output device1430, storage 1440, and communication device 1460.

Input device 1420 can be any suitable device that provides input, suchas a touch screen or monitor, keyboard, mouse, or voice-recognitiondevice. Output device 1430 can be any suitable device that providesoutput, such as a touch screen, monitor, printer, disk drive, orspeaker.

Storage 1440 can be any suitable device that provides storage, such asan electrical, magnetic, or optical memory, including a RAM, cache, harddrive, CD-ROM drive, tape drive, or removable storage disk.Communication device 1460 can include any suitable device capable oftransmitting and receiving signals over a network, such as a networkinterface chip or card. The components of the computer can be connectedin any suitable manner, such as via a physical bus or wirelessly.Storage 1440 can be a non-transitory computer-readable storage mediumcomprising one or more programs, which, when executed by one or moreprocessors, such as processor 1410, cause the one or more processors toexecute methods described herein, such as all or part of any one or moreof methods 1000, 1200, and 1300 described above with respect to FIGS.10, 12, and 13 .

Software 1450, which can be stored in storage 1440 and executed byprocessor 1410, can include, for example, the programming that embodiesthe functionality of the present disclosure (e.g., as embodied in thesystems, computers, servers, and/or devices as described above).Software 1450 can be implemented and executed on a combination ofservers such as application servers and database servers.

Software 1450 can also be stored and/or transported within anycomputer-readable storage medium for use by or in connection with aninstruction execution system, apparatus, or device, such as thosedescribed above, that can fetch and execute instructions associated withthe software from the instruction execution system, apparatus, ordevice. In the context of this disclosure, a computer-readable storagemedium can be any medium, such as storage 1440, that can contain orstore programming for use by or in connection with an instructionexecution system, apparatus, or device.

Software 1450 can also be propagated within any transport medium for useby or in connection with an instruction execution system, apparatus, ordevice, such as those described above, that can fetch and executeinstructions associated with the software from the instruction executionsystem, apparatus, or device. In the context of this disclosure, atransport medium can be any medium that can communicate, propagate, ortransport programming for use by or in connection with an instructionexecution system, apparatus, or device. The transport-readable mediumcan include, but is not limited to, an electronic, magnetic, optical,electromagnetic, or infrared wired or wireless propagation medium.

Computer 1400 may be connected to a network, which can be any suitabletype of interconnected communication system. The network can implementany suitable communications protocol and can be secured by any suitablesecurity protocol. The network can comprise network links of anysuitable arrangement that can implement the transmission and receptionof network signals, such as wireless network connections, T1 or T3lines, cable networks, DSL, or telephone lines.

Computer 1400 can implement any operating system suitable for operatingon the network. Software 1450 can be written in any suitable programminglanguage, such as C, C++, Java, or Python. In various aspects,application software embodying the functionality of the presentdisclosure can be deployed in different configurations, such as in aclient/server arrangement or through a Web browser as a Web-basedapplication or Web service, for example.

The foregoing description, for the purpose of explanation, has beendescribed with reference to specific aspects. However, the illustrativediscussions above are not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The aspects werechosen and described in order to best explain the principles of thetechniques and their practical applications. Others skilled in the artare thereby enabled to best utilize the techniques and various aspectswith various modifications as are suited to the particular usecontemplated.

Although the disclosure and examples have been fully described withreference to the accompanying figures, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosure and examples as defined bythe claims. Finally, the entire disclosure of the patents andpublications referred to in this application are hereby incorporatedherein by reference.

1. An image capture and display system for tissue sample tracking,comprising: a handheld image capture and pointing device; a displayassembly configured to display one or more images captured by thehandheld device; and one or more processors configured to cause thesystem to: capture, by the image capture and pointing device, an image;display, by a display of the display assembly, the image; while theimage is displayed by the display, determine a location on the displayassembly at which the handheld device is aimed, wherein the location onthe display corresponds to a region in the displayed image, wherein theregion in the displayed image comprises a target tissue area; and inresponse to detecting a user input and determining the location on thedisplay assembly, generate and store annotation data associated with thetarget tissue area.
 2. The system of claim 1, wherein the annotationdata comprises data indicating a region of interest in the image.
 3. Thesystem of claim 1, wherein the annotation data comprises a label for thetarget tissue area generated based on the user input.
 4. The system ofclaim 1, wherein the annotation data comprises time-stamp data generatedbased on a time-stamp of the image displayed on the display assembly. 5.The system of claim 1, wherein the annotation data comprises a uniqueidentifier automatically generated by the system.
 6. The system of claim1, wherein the annotation data is stored in association with a tissuesample taken from the target tissue area.
 7. The system of claim 1,wherein the annotation data is stored in association with pathologyresults data associated with the tissue sample taken from the targettissue area.
 8. The system of claim 1, wherein: the image is captured bythe image capture and pointing device, as part of a first time series ofvideo frames of the target tissue in a first spectrum; and the one ormore processors cause the system to: capture, by the image capture andpointing device, a second time series of video frames of the targettissue in a second spectrum; and in response to detecting a user inputand in accordance with the determined location on the display assembly,identify a region of interest in one or more frames of the second seriesof video frames and apply an image analysis algorithm to the region ofinterest in the one or more frames of the second series of video frames.9. A non-transitory computer-readable storage medium for image captureand display for tissue sample tracking, the non-transitorycomputer-readable storage medium storing instructions configured to beexecuted by one or more processors of a system for image capture anddisplay for tissue sample tracking comprising a handheld image captureand pointing device and a display assembly configured to display one ormore images captured by the handheld device, wherein executing theinstructions causes the system to: capture, by the image capture andpointing device, an image; display, by a display of the displayassembly, the image; while the image is displayed by the display,determine a location on the display assembly at which the handhelddevice is aimed, wherein the location on the display corresponds to aregion in the displayed image, wherein the region in the displayed imagecomprises a target tissue area; and in response to detecting a userinput and determining the location on the display assembly, generate andstore annotation data associated with the target tissue area.
 10. Amethod for image capture and display for tissue sample tracking, themethod performed at a system for image capture and display for tissuesample tracking comprising a handheld image capture and pointing device,a display assembly configured to display one or more images captured bythe handheld device, and one or more processors, the method comprising:capturing, by the image capture and pointing device, an image;displaying, by a display of the display assembly, the image; while theimage is displayed by the display, determining a location on the displayassembly at which the handheld device is aimed, wherein the location onthe display corresponds to a region in the displayed image, wherein theregion in the displayed image comprises a target tissue area; and inresponse to detecting a user input and determining the location on thedisplay assembly, generating and storing annotation data associated withthe target tissue area.